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Preface 


This  paper  was  prepared  by  the  Institute  for  Defense  Analyses  (IDA)  for  the  Joint  Staff  s 
Wargaming,  Simulation  and  Assessment  Directorate  (J-8)  under  the  task  order  Joint  Battle 
Center  Experimentation  and  Assessment.  The  study  was  conducted  in  response  to  the  task 
objective  “to  assist  the  [Joint  Battle  Center]  JBC  in  the  design  and  execution  of  experiments 
which  explore  and  evaluate  new  warfighting  concepts  and  supporting  C4ISR  technology.” 

The  purpose  of  this  paper  is  to  advance  and  fuel  the  debate  on  the  meaning  of  joint 
experimentation.  Experimentation  supports  the  process  of  innovation,  a  process  that  can 
lead  to  evolutionary  or  revolutionary  change.  Our  goal  here  is  to  provide  a  point  of 
departure  that  includes  a  useful  framework,  a  lexicon,  and  definitions  for  the  joint 
experimentation  community.  Most  of  the  work  is  oriented  towards  the  idea  of  the  military 
experiment  and  is  equally  suited  to  Service  or  Joint  applications.  The  material  in  this 
document  is  an  extract  of  a  much  larger  body  of  work  conducted  for  the  same  sponsor. 

Our  J-8  sponsor,  Mr.  Vincent  P.  Roske,  Jr.,  has  graciously  provided  the  support  and 
latitude  to  pursue  this  topic  for  the  larger  community.  Mr.  William  “Andy”  Rumbaugh  of 
the  Joint  Battle  Center  read  early  drafts  and  provided  thoughtful  comments  and 
encouragement. 

Several  research  staff  members  within  IDA  took  the  time  to  review  the  many  drafts  and 
provide  useful  guidance  in  the  development  of  the  work.  They  are  Major  General  Larry  D. 
Budge,  USA  (ret.);  Dr.  William  J.  Hurley;  Dr.  Richard  J.  Ivanetich;  Lieutenant  General 
Peter  Kind,  USA  (ret.);  Colonel  Steve  McNamara,  USAF;  Colonel  Tom  O’Leeiry,  USMC; 
Colonel  Mike  Starry,  USA  (ret.);  Lieutenant  General  Paul  K.  Van  Riper,  USMC  (ret.);  and 
Mr.  Phillip  J.  Walsh. 
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Executive  Summary 


The  Department  of  Defense  has  recently  embarked  on  a  mission  to  experiment  with  new 
warfighting  concepts  and  capabilities.  It  appears  that  the  experiment  practiced  in  college 
physics  labs  is  serving  as  the  de  facto  model  for  the  military  experiment.  However,  the 
hypothesis-testing  science  experiment  has  little  applicability  to  military  experimentation. 
Three  alternative  experimental  forms  better  suited  to  military  experimentation  are  offered  in 
this  paper:  (1)  the  field  research  method,  (2)  applied  research  and  exploratory  development, 
and  (3)  applied  social  research.  (Figure  ES-1  depicts  an  overview.) 

The  field  research  method  is 
employed  in  the  discovery  phase 
of  scientific  inquiry  to  determine 
how  a  phenomenon  works  and  to 
generate  plausible  explanations  for 
why  it  works.  Field  research  is 
often  conducted  in  an  iterative 
collect-interpret-collect  fashion. 

Heuristics  are  questions  that  guide 
the  investigation  and  evolve  at 
each  iteration. 

In  contrast,  inventors  and 
engineers  practice  applied  research  and  exploratory  development  that  culminates  in  a 
prototype  product.  Practitioners  often  apply  theory  and  intuition  through  a  trial-and-error 
procedure  that  terminates  when  a  performance  goal  is  reached. 

Social  scientists  practice  applied  social  research  to  inform  public  policy  makers.  One 
of  the  important  functions  of  applied  social  research  is  the  measurement  of  a  program’s 
effectiveness,  called  simple  effects  measurement.  Another  function,  relative  effects 
measurement,  compares  the  effectiveness  of  several  programs. 

All  of  these  research  methods  employ  experiments.  A  fourth  experimental  approach 
mentioned  in  this  paper,  the  hypothesis-testing  experiment,  is  the  least  applicable  to  military 
experiments.  In  contrast,  the  field  research  method  supports  the  discovery  phase  of  military 
experimentation  necessary  to  achieve  grand  innovations.  Applied  research  and  exploratory 
development  is  appropriate  for  inventing  a  solution.  Applied  social  research  is  most 
applicable  to  guide  resource  allocation  decisions. 
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Figure  ES-1.  Overview 
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The  word  experiment  has  several  meanings.  To  each  community,  its  own  definition  is 
correct.  However,  the  definitions  are  correct  only  because  they  are  useful  to  that 
conununity.  The  military  community  should  borrow  and  adapt  those  definitions  that 
produce  the  desired  results  of  military  experimentation.  The  following  simple  definition  is 
offered. 

military  experimentation,  n.  The  process  of  exploring  innovative  methods  of 
operation,  especially  to  assess  their  feasibility,  evaluate  their  utility,  or  determine 
their  limits. 

Having  a  working  definition  is  useful,  but  not  as  important  as  understanding  that 
military  experimentation  is  a  learning  process  built  on  discovery  and  invention  far  more 
than  it  is  a  hypothesis-testing  process  that  verifies  theoretical  cause-and-effect  relationships. 

The  word  joint  is  used  in  a  variety  of  contexts.  The  meaning  derived  from  each  context 
is  legitimate,  but  the  variety  of  meanings  makes  it  difficult— perhaps  impossible — ^to 
determine  exactly  what  “joint  experimentation”  might  be.  Some  examples  of  common 
usage  follow. 

•  Joint:  The  Joint  Staff,  unified  combatant  commands,  subordinate  unified 
commands,  joint  task  force  headquarters. 

•  Joint:  The  strategic  and  operational  responsibilities  of  joint  commands. 

•  Joint:  Multiple  Service  or  all  Service,  i.e.,  a  committee  of  Service  representatives 
with  no  one  in  charge. 

•  Joint:  An  issue  of  interest  to  more  than  one  Service,  e.g.,  infantry  weapons  (or 
tactics)  to  the  Army  and  Marine  Corps,  or  strike  fighters  (or  tactics)  to  the  Air  Force, 
Navy,  and  Marine  Corps. 

•  Joint:  Service  interoperability,  e.g.,  interoperability  of  Army  and  Marine  Corps 
ground  communications  or  of  Air  Force  and  naval  aircraft  communications. 

Based  on  statements  made  by  senior  leadership  within  the  executive  and  legislative 
branches,  the  following  definition  is  offered  for  joint  experimentation. 

joint  experimentation,  n.  Military  experimentation  conducted  to  (a)  develop  a 
Service-specific  military  capability  for  employment  by  the  Joint  Force 
Commander,  (b)  develop  a  military  capability  without  regard  to  current  roles-and- 
missions  boundaries,  or  (c)  to  allocate  resources  across  Service  programs. 
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What  Does  “Military  Experimentation”  Really  Mean? 


The  Department  of  Defense  recently  established  experimentation  as  a  new  military 
mission.'  What  does  experimentation  mean  in  a  military  sense?  What  does  a  proper 
military  experiment  most  resemble,  and  what  are  its  elements? 

The  physical,  social,  and  life  sciences  each  use  words  like  experiment  and  hypothesis 
differently.  Each  community  of  scientists  considers  its  own  definitions  correct  because 
these  same  definitions  are  useful  for  their  purposes.  Military  experimenters  should  borrow, 
adapt,  and  develop  definitions  that  best  support  their  purposes.  This  paper  examines  various 
approaches  in  search  of  meanings  that  are  useful  for  the  purposes  of  peacetime  military 
experimentation. 

Scientists  involved  in  the  pursuit  of  new  knowledge  employ  the  methods  of  scientific 
inquiry.  The  earliest  phases  of  scientific  inquiry  are  about  exploration  and  discovery  and 
often  result  in  plausible  theories;  the  latter  phases  are  about  testing  and  verifying  those 
theories.  The  field  research  method  of  scientific  inquiry  employs  the  heuristically  guided 
discovery  experiment  and  the  experimental  research  method  is  based  on  the  hypothesis¬ 
testing  verification  experiment. 

Inventors  and  engineers,  rather  than  pursuing  new  knowledge,  apply  available 
knowledge  to  develop  products  for  patent  or  for  market.  They  often  apply  a  trial-and-error 
procedure  in  a  goal  seeking  invention  experiment.  Social  scientists  and  policy  analysts  use  a 
fourth  type  of  experiment,  the  performance  measuring  experiment,  to  monitor  existing 
program  performance  and  predict  new  program  performance  to  inform  resource  allocation 
decisions. 


The  Purposes  of  Military  Experimentation 

Calls  to  transform  the  United  States  military  come  from  congressional  leaders,  senior 
Defense  officials,  the  Joint  Chiefs  of  Staff,  and  the  academic  community.  Many  believe  we 
are  witnessing  a  “revolution  in  military  affairs”  similar  to  that  which  occurred  between 
World  Wars  I  and  H,  and  call  for  experimentation  as  a  means  to  foster  innovation.  But  a 
strategy  for  pursuing  a  revolution  in  military  affairs,  like  any  strategy,  must  subordinate 
means  to  ends.  Neither  innovation  nor  experimentation  is  the  end  we  are  pursuing. 

The  end  to  be  continuously  maintained  is  a  military  instrument  in  the  hands  of  national 
leaders — an  instrument  that  is  suited  to  the  pursuit  of  our  national  interests,  superior  in 
military  effectiveness  to  our  adversaries,  and  sustainable  in  peace  and  war.  Evolutionary 
and  revolutionary  improvements  in  effectiveness  and  efficiency  can  both  contribute  to  this 
end;  however,  some  assert  that  revolutionary  change  is  necessary  or  at  least  desirable.  The 


'  Department  of  Defense  News  Release  252-98,  U.S.  Atlantic  Command  Designated  Executive  Agent  for  Joint 
Warfighting  Experimentation,  May  21,  1998. 
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end,  therefore,  is  a  transformed  military  instrument  containing  at  least  some  elements  that 
represent  a  grand  departure  from  current  capabilities.  But  the  end  is  elusive,  continually 
pursued,  and  never  reached. 

Pursuing  this  end  poses  a  host  of  difficult  questions.  What  will  be  the  future  threats  to 
our  national  interests?  What  will  be  our  national  security  strategy?  How  will  we  organize 
for  and  conduct  military  operations?  What  new  military  capabilities  should  we  develop? 
Which  current  capabilities  should  we  embrace  and  which  will  we  abandon?  How  should  we 
allocate  scarce  resources  among  a  seemingly  endless  array  of  competing  alternatives?  The 
process  underwriting  the  transformation  strategy  is,  then,  one  of  acquiring  the  knowledge 
necessary  to  answer  these  questions.  A  fundamental  purpose  of  military  experimentation  is 
the  acquisition  of  knowledge  to  guide  decisions  about  an  uncertain  future. 

Every  experiment  should  be  designed  to  produce  information  that  eventually  informs  a 
decision.  But  different  decisions  require  different  information.  What  information  should  be 
generated  and  gathered?  While  data  collection  plans  are  dictated  by  each  specific 
experiment,  certain  useful  generalizations  can  be  made,  depending  on  the  type  of 
experiment  and  its  purpose.  Figure  1  suggests  several  objects  of  experimentation. 


Discovery 


Objects  Invention 


require 


Figure  1.  Experimentation  Supports  Both  Discovery  and  Invention 
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Volumes  have  been  written  recently  on  military  innovation,  particularly  in  the  period 
preceding  World  War  H?  In  this  body  of  literature,  it  is  commonly  asserted  that  during 
times  of  relative  peace,  methods  of  employment  (strategies,  doctrines,  tactics,  techniques, 
procedures,  and  organizational  arrangements)  do  not  keep  pace  with  technological  advances. 
The  inability  of  either  side  to  fully  exploit  technological  advances  in  World  War  I,  for 
example,  is  offered  as  one  explanation  for  the  sluggish  and  costly  stalemate  that  ensued.  In 
World  War  n,  both  sides  incorporated  the  significant  technological  advances  of  the  interwar 
period,  and  then  forced  a  dramatic  acceleration  as  new  methods  of  emplo5mient  were 
attempted  and  rapidly  adopted,  adapted,  or  discarded  in  response  to  combat  outcomes. 

Actual  combat  is  the  most  demanding  and  unforgiving  laboratory  for  military 
experimentation  with  tools  and  techniques.  There  is  survival  for  those  who  succeed  and 
crushing  defeat  for  those  who  fail.  An  important  goal  of  military  experimentation  in 
peacetime,  then,  must  be  to  develop  employment  methods  that  keep  pace  with  and  fully 
exploit  the  ever-changing  tools  of  war.  The  results  may  vary  according  to  the  magnitude  of 
the  departure  from  past  practices:  the  so-called  revolutionary  and  evolutionary  innovations. 
The  role  played  by  the  military  experiment  varies  considerably,  depending  on  the  magnitude 
of  innovation  attempted. 

The  fundamental  value  of  a  military  experiment  is  that  it  provides  the  opportunity  to 
observe  military  phenomena  empirically — to  learn  by  doing  without  the  attendant  costs  of 
war.  Military  experimentation  is  therefore  a  process  of  exploration  and  discovery.  Any 
experiment  ought  to  permit  the  observation  of  military  phenomena  to  support  the  higher 
purpose  of  keeping  tactics,  techniques,  and  procedures  abreast  of  technological  advances — 
exploiting  technology  through  methods  of  employment. 


Research  Methods 

It  is  useful  to  make  a  distinction  between  basic  research  and  applied  research.  Basic 
research  is  undertaken  in  pursuit  of  knowledge  for  the  sake  of  the  knowledge  itself. 
Scientists  employ  the  methods  of  scientific  inquiry  to  develop  a  body  of  knowledge  or 
theory.  Applied  research,  on  the  other  hand,  applies  knowledge  to  a  problem  of  practical 
significance,  perhaps  developing  new  knowledge  along  the  way.  This  form  of  research  is 
commonly  practiced  by  inventors,  engineers,  and  public  policy  analysts,  albeit  differently. 
The  social  scientist  typically  is  concerned  with  understanding  social  problems  and  informing 
public  policy.  In  contrast,  elements  of  the  private  sector  employ  applied  research  in  the 
early  stages  of  bringing  a  product  to  market. 


^  See,  for  example,  Military  Innovation  in  the  Interwar  Period,  W.  Murray  and  A.R.  Millett,  eds.,  (New  York, 
NY:  Cambridge  University  Press,  1 996). 
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The  Methods  of  Basic  Research  and  Scientific  Inquiry 

An  initial  finding  in  the  research  leading  to  this  paper  is  that  the  scientific  experiment  as 
taught  in  undergraduate  physics  and  chemistry  is  being  taken  by  many  as  the  model  for 
military  experimentation.  Much  of  what  is  written  about  military  experimentation  includes 
discussion  of  hypothesis  formulation  and  testing,  control  of  independent  variables,  and  trials 
of  sufficient  numbers  to  provide  statistical  significance  and  inference.  We  believe  this  type 
of  seientific  experiment  has  little  applicability  or  utility  to  those  conducting  military 
experiments,  or  to  the  decision  makers  who  must  act  on  their  outcomes. 
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Figure  2.  Knowledge  Acquisition  and  Scientific  Inquiry’ 


Figure  2  above  depicts  scientific  inquiry  as  but  one  of  the  methods  of  knowledge 
acquisition.  Scientific  inquiry  has  two  phases — discovery  and  verification — with  the 
preponderance  of  effort  devoted  to  the  first.  The  lower  half  of  the  figure  adequately 
describes  the  methods  of  basic  research. 


^  Adapted  from  Experimental  Methodology,  l'^  ed.  by  L.B.  Christensen  (Boston:  Allyn  and  Bacon,  1997). 
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The  discovery  phase  of  scientific  inquiry  has  three  aims.  The  first  is  to  identify  a 
problem  worth  solving.  Problems  abound,  but  identifying  problems  that  are  worth  solving 
and  amenable  to  solution  by  scientific  methods  eliminates  many  candidates  from 
consideration.  The  second  aim  is  to  describe  phenomena  accurately.  A  good  description 
portrays  some  phenomenon  as  it  is  observed,  identifies  the  variables  at  work  in  the 
phenomenon,  and  identifies  the  degree  to  which  each  variable  participates.  The  third  aim  is 
to  develop  one  or  more  plausible  explanations  of  why  the  phenomenon  occurs,  i.e.,  what 
causes  it. 

The  objectives  of  the  verification  phase  are  different  from  the  discovery  phase. 
Alternative  explanations  are  tested  using  scientific  methods  to  confirm  causality,  test  theory, 
and  establish  predictability.  Successful  explanations,  or  theories,  allow  the  scientist  to 
predict  an  outcome  based  on  the  values  of  independent  variables — ^to  accurately  anticipate 
an  event  prior  to  its  actual  occurrence.  Control,  in  this  context,  refers  to  the  ability  to 
control  outcomes  (dependent  variables)  by  specific  interventions  (manipulation  of 
independent  variables),  and  implies  the  ability  to  generalize  from  the  experiments. 

In  addition  to  a  number  of  objectives,  there  are  several  styles  of  scientific  inquiry.  The 
hypothesis-testing  scientific  experiment  is  part  of  the  experimental  research  method  and  is 
used  to  support  the  verification  phase.  Thus,  a  focus  on  the  hypothesis-testing  experiment 
excludes  most  methods  of  knowledge  acquisition,  the  discovery  phase  so  critical  to 
scientific  inquiry,  most  objectives  of  scientific  inquiry,  and  most  styles  of  research.  We 
believe  this  exclusion,  which  is  implicit  in  many  current  military  experimentation 
documents,  is  unintentional  and  counterproductive. 

The  field  research  method  is  far  better  suited  to  the  discovery  phase  than  the 
experimental  research  method  with  its  hypothesis-testing  experiments.  In  field  research  as 
practiced  in  the  social  sciences,  the  scientist  poses  one  or  more  questions  for  which  he  or 
she  has  no  confident  answers,  and  a  case  study  (with  embedded  experiment)  is  proposed  that 
might  provide  answers.  The  experiment  might  be  based  on  already-recorded  empirical  data, 
such  as  training  data  or  analysis  of  actual  military  conflicts.  It  may  be  based  on  the  results 
of  simulation,  an  experiment  embedded  in  real-world  operations,  or  a  field  experiment 
might  be  designed  specifically  to  generate  the  data.  In  any  case,  the  experiment  is  designed 
to  provide  answers  to  specific  questions.  A  good  experiment  might  generate  more  questions 
than  it  answers,  and  likely  will  require  that  a  new  case  study  be  designed  to  answer  the  new 
questions  posed.  These  constantly  evolving  questions  are  called  heuristics,  and  this 
approach  to  the  acquisition  of  knowledge  is  called  heuristically  guided  investigation. 

This  formulation — experimentation  as  a  heuristically  guided  investigation  to  support 
discovery — is  at  odds  with  the  apparent  definition  provided  by  the  hypothesis-testing 
experiment  employed  to  verify  theories  as  part  of  the  process  of  scientific  inquiry. 

The  Methods  of  Applied  Research  and  Exploratory  Development 

Scientists  conducting  basic  research  are  not  the  only  community  that  experiments. 
Engineers  and  inventors  also  experiment  but  use  a  different  procedure.  While  scientists 
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tend  to  pursue  new  knowledge  through  basic  research,  inventors  and  engineers  pursue 
solutions  to  practical  problems  through  applied  research,  often  based  on  trial  and  error. 
From  this  distinction,  one  might  correctly  conclude  that  some  military  experiments  would 
be  more  like  those  of  engineers  than  of  scientists. 

Applied  research  and  exploratory  development  do  in  fact  offer  useful  analogies  for 
military  experimentation.  Those  involved  in  applied  research  and  exploratory  development 
employ  an  iterative  trial-and-error  procedure  that  terminates  when  the  performance  goal  is 
achieved.  To  prove  a  concept,  applied  research  may  conclude  with  the  exploratory 
development  of  a  prototype  product.  Bringing  a  new  military  capability  into  being  bears 
considerable  resemblance  to  bringing  an  innovative  product  to  market.  Clearly,  this  type  of 
experimentation  has  military  application. 

The  Methods  of  Applied  Social  Research 

Another  type  of  decision  that  requires  the  acquisition  of  knowledge  is  concerned  with 
allocating  scarce  resources  among  competing  programs.  Decision  makers  need  experiments 
formulated  to  provide  the  necessary  information.  Experiments  could  be  designed  to 
measure  the  potential  utility  of  proposed  capabilities  or  the  actual  utility  of  newly  developed 
capabilities.  Social  scientists  are  well  practiced  in  this  type  of  experiment  in  their  attempts 
to  inform  government  actions. 

Public  policy  is  implemented  through  a  variety  of  programs.  Negative  symptoms  tend  to 
focus  public  debate,  and  policy  objectives  are  formulated  to  counter  or  reverse  symptoms. 
Based  on  theoretical  cause-and-effect  relationships,  interventions  are  proposed  to  achieve 
policy  objectives.  Programs  are  funded  that  implement  the  interventions.  Empirical 
evidence  is  collected  from  ongoing  programs. 


Table  1.  Applied  Social  Research  Questions' 


Type 

Examples 

descriptive 

What  methods  are  being  used  to  treat  drug  addiction? 

What  percentage  of  addicts  receives  treatment? 

normative 

Are  treatment  programs  performing  adequately? 

How  well  managed  are  treatment  programs? 

correlative 

Is  there  a  correlation  between  arrest  and  treatment? 

What  is  the  relationship  between  treatment  and  subsequent  arrest? 

simple  ejfects 

Does  participation  in  a  treatment  program  lower  recidivism? 

relative  effects 

Which  type  of  treatment  is  most  effective  in  lowering  recidivism? 

'  Derived  from  Applied  Research  Designs:  A  Practical  Guide,  T.E.  Hedrick  et  al.  (Thousand  Oaks,  CA:  SAGE 
Publications,  Inc.,  1993). 
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Applied  social  research  questions  tend  to  be  descriptive,  normative,  correlative,  or 
effects  related.  Table  1  above  provides  examples  of  the  kinds  of  questions  addressed  by 
social  scientists.  Examples  are  taken  from  the  domain  of  national  dmg  policy.  Capping  this 
set  of  questions  is  “What  is  the  best  mix  of  programs?”  Collectively,  these  questions  are 
quite  similar  to  the  questions  that  drive  military  force  structure  analysis. 

Summary  of  Research  Methods 

In  short,  there  are  several  communities  whose  methods  offer  analogies  for  military 
experiments  (see  Table  2  below).  Basic  research  methods  used  in  the  pursuit  of  knowledge 
include  both  heuristically  guided  discovery  and  hypothesis-testing  verification.  Inventors 
and  engineers  use  applied  research  and  exploratory  development  to  address  practical 
problems,  employing  a  goal-seeking,  trial-and-error  procedure.  And  social  scientists  use 
applied  social  research  methods  in  measuring  the  effects  (intended  or  unintended)  of 
government  interventions  (planned  or  implemented). 

Table  2.  Broad  Categories  of  Experimental  Forms 


Driving  Principle 

Dominant  Procedure 

Purpose 

heuristic  guiding 

exploration/investigation 

discovery 

hypothesis-testing 

multiple,  controlled  trials 

verification 

goal  seeking 

trial  and  error/honing 

invention 

effects  measuring 

operational  evaluation 

resource  allocation 

Adapting  Experiments  to  Military  Purposes 

The  previous  section  surveyed  the  experimental  methods  of  non-military  communities 
for  useful  analogies.  This  section  adapts  and  applies  those  analogies  to  the  purposes  of 
military  experimentation.  Three  broad  categories  are  considered:  measuring  the  effects  of 
proposed  or  newly  fielded  capabilities,  improving  current  practices,  and  innovating  in  grand 
fashion. 

Measuring  Effects 

There  is  no  shortage  of  ideas,  but  which  ideas  offer  the  greatest  payoff?  Fiscal  realities 
dictate  that  not  all  meritorious  proposals  will  be  pursued.  The  surest  and  most  accurate  way 
to  evaluate  a  capability’s  actual  effectiveness  would  be  to  build  the  systems  required  to 
implement  it,  try  them,  and  then  decide  whether  to  put  the  systems  into  production  and 
fielding.  For  any  but  the  simplest  systems,  this  approach  is  prohibitively  expensive  in  terms 
of  both  time  and  money.  A  more  affordable  but  less  accurate  approach  is  to  build  a  model 
(in  the  loosest  sense)  of  the  necessary  systems  and  capability,  measure  the  capability’s 
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potential  effectiveness,  and  then  decide  whether  to  invest  in  the  research  and  development 
necessary  to  field  the  supporting  systems  and  the  desired  capability. 

These  two  approaches  require  different  tools.  Measuring  effects  in  the  field  can  only  be 
conducted  with  real  equipment  and  forces;  the  demand  here  is  for  instrumented  ranges  and 
at  least  production-representative  systems.  Measuring  actual  effects  is  most  closely 
approximated  by  traditional  operational  test  and  evaluation.  Measuring  potential  effects 
employs  wargames,  analytic  models,  human-in-the-loop  simulations,  and  field  exercises 
using  real  equipment  and  surrogates.  All  of  the  supporting  tools  have  limitations — ^but  all 
have  a  contribution  to  make. 

Experiments  conducted  as  field  exercises,  particularly  large  ones,  are  not  repeatable  in 
the  rigorous  sense  and,  in  any  case,  are  too  costly  to  repeat.  They  appear  to  enjoy  greater 
credibility  than  computer  simulations  but  less  credibility  than  empirical  evidence  derived 
from  actual  warfare.  Computer  simulations  allow  the  large  numbers  of  trials  necessary  to 
generalize,  but  suffer  from  lower  credibility.  The  results  of  computer  simulations  are  rarely 
referred  to  as  empirical  evidence.  Some  combination  of  these  tools — capitalizing  on  the 
strengths  of  each  while  offsetting  their  respective  weaknesses — ^is  probably  the  best 
approach. 

The  purpose  of  this  type  of  experimentation  is  to  measure  performance.  The  emphasis  is 
on  determining  performance  measures — designing  an  experiment  that  generates  the 
necessary  data,  and  collecting  and  interpreting  the  data — and  not  on  testing  hypotheses 
derived  from  an  underlying  theory  of  the  phenomenon. 

Improving  Performance  and  Overcoming  Deficiencies 

All  current  military  capabilities  can  be  improved,  but  some  deficiencies  are  more  glaring 
than  others.  Poor  performance  in  combat  or  in  training  may  point  out  a  deficiency;  or  a  new 
weapon  or  information  system  dropped  into  current  organizations  and  doctrine  may 
introduce  an  opportunity  to  improve  performance.  If  a  capability  already  performs  close  to 
its  desired  goal,  then  we  are  “honing.”  Honing  takes  place  naturally  as  part  of  daily 
operations  and  training.  If  a  capability  is  far  from  its  desired  goal,  then  we  are  engaged 
more  in  an  invention  process.  The  distinction  is  the  degree  of  freedom  applied  to  varying 
the  independent  variables. 

For  this  type  of  military  experiment,  methods  of  emplo5ment  serve  as  independent 
variables  and  explicit  performance  measures  serve  as  the  dependent  variables.  A  trial-and- 
error  procedure  is  applied  until  a  specified  goal  is  reached.  The  experiment  design  must 
state  measurable  goals  and  provide  a  data  collection  plan  accordingly;  stating  a  hypothesis 
serves  no  purpose. 

Reducing  fratricide  in  close  operations  is  a  good  example.  We  already  know  how  to 
conduct  coordinated  air-ground  operations  in  the  close  fight,  but  experiments  might  be 
conducted  that  vary  tactics,  techniques,  and  procedures  (independent  variables)  and  measure 
the  effects  on  fratricide,  lethality,  and  survivability  (dependent  variables). 
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Innovating  in  Grand  Fashion 

Grand  innovations  represent  a  significant  departure  from  previous  practice.  With  grand 
innovation,  one  thing  we  can  state  unequivocally  is  that  we  don ’t  know  what  we  don’t  know. 
An  essential  purpose  of  experimentation  is  to  provide  the  opportunity  to  observe  military 
phenomena,  to  learn  something  about  that  which  we  don’t  know,  and  to  generate  questions. 
We  may  have  a  difficult  time  identifying  the  questions  for  the  initial  experiment  in  a 
heuristically  guided  investigation.  Thus,  the  first  experiment  might  be  conducted  to 
generate  an  initial  round  of  questions.  Other  experiments  might  follow  that  answer  those 
questions  and  generate  new  ones.  Heuristics  are  needed,  not  hypotheses. 

At  some  point,  the  heuristically  guided  investigation  gives  way  to  an  invention  process 
and  heuristics  give  way  to  measurable  goals.  The  invention  process  is  goal  seeking.  Only 
after  a  capability  is  “invented”  should  it  be  subjected  to  evaluation.  The  evaluation  phase  is 
most  analogous  to  the  effects-measuring  experiment  of  applied  social  research.  Once 
implemented  and  fielded,  the  process  of  improving  capabilities  takes  over.  Grand 
innovation  requires  the  entire  spectrum  of  experimentation  types  described  in  this  paper. 

Even  after  a  new  grand  capability  is  demonstrated,  it  cannot  simply  be  handed  over  to 
operational  conunands,  nor  can  operational  commands  immediately  begin  to  train  to 
standards  with  the  new  capability.  They  must  have  the  opportunity  to  learn  how  and  when 
to  employ  the  capability  and  to  learn  its  limits. 

The  scholarly  examination  presented  in  America’s  First  Battles,  1776-1965,^  leads 
quickly  to  the  conclusion  that  “more  glaring  than  poorly  trained  troops  as  a  first-battle 
problem  is  the  weakness  of  command-and-control.”  The  authors  go  on  to  attribute  this 
weakness  to  “inadequate  preparation  of  commanders  and  staffs  for  the  real  world  of 
combat.”  Military  trainers  have  taken  these  admonitions  to  heart  and  have  structured 
training  programs  to  provide  commanders  and  staffs  with  “first  battle”  experience  in 
peacetime  and  in  a  benign  environment.  Training  is  the  correct  method  for  teaching  leaders 
and  organizations  how  to  employ  proven  methods,  but  training  is  inadequate  to  the  task  of 
developing  and  learning  how  to  employ  methods  unproven  in  combat.  Experimentation 
must  be  the  precursor  to  training. 

In  wartime,  technological  advances  accelerate  as  more  resources  are  allocated  to 
research  and  development,  and  exploitation  methods  are  hastily  developed  and  honed  in 
combat.  Necessity  is  truly  the  mother  of  invention.  Peacetime  military  experimentation 
must  take  the  place  of  wartime  experience;  otherwise,  first  battles  will  provide  the  learning 
environment  at  unacceptably  high  cost. 


^  America’s  First  Battles,  } 776- 1965,  C.E.  Heller  and  W.A.  Stofft,  eds.,  (Lawrence,  Kansas:  University  Press 
of  Kansas,  1986). 
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What’s  Different  about  Joint  Experimentation? 

The  discussion  so  far  is  equally  applicable  to  experimentation  within  a  single  Service  as 
it  is  to  joint  experimentation.  It  is  the  purpose  to  which  the  results  of  experiments  are  put 
that  distinguish  joint  from  Service  experimentation.  Joint  experiments  are  conducted  for 
one  or  more  of  the  following  three  purposes. 

•  Developing  Service-specific  military  capability  for  employment  by  the  Joint  Force 
Commander. 

•  Developing  military  capability  without  regard  to  current  roles-and-missions 
boundaries. 

•  Allocating  resources  across  Service  programs. 

It  is  difficult  to  imagine  designing  a  single  experiment  that  could  achieve  all  these 
purposes  simultaneously.  More  to  the  point,  various  decision  makers  within  the  joint 
experimentation  community  have  significantly  different  objectives  in  mind.  All  these 
decision  makers  have  legitimate  information  needs,  and  it  is  unlikely  that  they  will  all  be 
equally  satisfied  by  a  single  experiment  or  even  a  single  type  of  experiment. 

Depending  on  the  magnitude  of  departure  from  current  practice,  developing  military 
capability  will  be  adequately  supported  by  the  methods  supporting  grand  innovation  or 
capability  improvement.  Given  that  the  Services  hold  the  purse,  who  will  develop  military 
capability  independent  of  current  Service  roles-and-missions  boundaries? 

Institutionally,  a  single  Service  may  be  able  to  decide  how  to  allocate  resources  across 
its  own  programs — that  is,  to  pick  winners  and  losers — ^by  conducting  evaluative 
experiments.  Deciding  how  to  allocate  resources  among  the  Services  is  another  matter.  The 
Services  that  must  pay  for  the  experiment  do  so  at  the  risk  of  losing  a  favored  program. 
Who  will  manage,  fund,  and  evaluate  cross-Service  experimentation? 

In  addition,  the  word  joint  is  used  in  a  variety  of  contexts.  The  meaning  derived  from 
each  context  is  legitimate,  but  the  variety  of  meanings  makes  it  difficult — ^perhaps 
impossible — to  determine  exactly  what  “joint  experimentation”  might  be.  Some  examples 
of  common  usage  are  listed  below  and  depicted  in  Figure  3. 

•  Joint:  The  Joint  Staff,  unified  combatant  commands,  subordinate  unified  commands, 
joint  task  force  headquarters. 

•  Joint:  The  strategic  and  operational  responsibilities  of  joint  commands. 

•  Joint:  Multiple  Service  or  all  Service,  i.e.,  a  committee  of  Service  representatives  with 
no  one  in  charge. 

•  Joint:  An  issue  of  interest  to  more  than  one  Service,  e.g.,  infantry  weapons  (or  tactics) 
to  the  Army  and  Marine  Corps,  or  strike  fighters  (or  tactics)  to  the  Air  Force,  Navy,  and 
Marine  Corps. 

•  Joint:  Service  interoperability,  e.g.,  interoperability  of  Army  and  Marine  Corps  ground 
communications  or  of  Air  Force  and  naval  aircraft  communications. 
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The  Joint  Force 
Commander  and  Staff 

Strategic  &  Operational 
Levels  of  War 

JTF  Component  HQs 

Service  system 
Interoperability 
Joint  (e.g.,  plane-to-tank) 

Force 

•  Weapon  systems  of  interest 
to  two  or  more  Services 
(e.g.,  Joint  Strike  Fighter) 

A  committee  of  Service 
representatives 

Joint  Force  (head  to  toe) 


Figure  3.  The  Many  Meanings  of  Joint 

No  amount  of  analysis  will  determine  which  of  these  definitions  should  apply  when  we 
talk  of  joint  experimentation.  However,  senior  decision  makers  in  the  legislative  and 
executive  branches  must  come  to  an  understanding  of  what  joint  means  in  joint 
experimentation.  In  all  likelihood,  no  single  definition  will  suffice.  Appropriate  definitions 
of  joint  experimentation,  as  suggested  by  the  above  taxonomy,  should  be  determined  case  by 
case.  As  objects  of  experimentation,  methods  of  employment  should  be  strongly  favored 
over  systems. 


Conclusion 

In  summary,  the  word  experiment  means  many  things  to  many  people.  To  each 
community  that  conducts  experiments,  its  own  definitions  are  correct.  However,  these 
definitions  are  correct  only  because  they  are  useful  to  that  community.  The  military 
community  need  not  select  one  definition.  Rather,  it  should  borrow,  adapt,  and  develop 
definitions  and  frameworks  that  produce  the  desired  results  of  military  experimentation. 
The  following  simple  definition  is  offered. 

military  experimentation,  n.  The  process  of  exploring  innovative  methods  of 
operation,  especially  to  assess  their  feasibility,  evaluate  their  utility,  or  determine  their 
limits. 

Of  course  the  actual  definition  is  not  as  useful  as  understanding  that  military 
experimentation  is  a  learning  process — one  which  supports  discovery  and  invention  as 
shown  in  Figure  1 — far  more  than  it  is  a  hypothesis-testing  process  that  verifies  theoretical 
cause-and-effect  relationships. 

Grand  innovation  is  best  supported  by  two  phases  of  experimentation.  The  first  requires 
experiments  that  support  discovery  and  exploration  that  culminate  in  greater  insight  and  in 
at  least  one  promising  solution  approach.  The  second  phase  requires  experiments  that 
support  invention  that  possibly  produce  a  prototype  capability.  By  contrast,  improving 
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existing  military  capability  requires  experiments  that  support  invention  and  honing  of 
performance.  Resource  allocation  decisions  may  be  required  to  field  innovative  capabilities, 
and  those  decisions  may  be  properly  supported  by  experiments  measuring  simple  and 
relative  effects. 

Based  on  statements  made  by  senior  leadership  within  the  executive  and  legislative 
branches,  the  following  definition  is  offered  for  joint  experimentation. 

joint  experimentation,  n.  Military  experimentation  conducted  to  (a)  develop  a 
Service-specific  military  capability  for  employment  by  the  Joint  Force 
Commander,  (b)  develop  a  military  capability  without  regard  to  current  roles-and- 
missions  boundaries,  or  (c)  to  allocate  resources  across  Service  programs. 

Military  and  joint  experimentation  are  important  topics  discussed  at  the  highest  levels  of 
government,  yet  it  is  not  at  all  clear  that  participants  in  the  discussion  are  talking  about  the 
same  things.  Agreement  on  the  meaning  of  these  important  terms  is  a  necessary  step  in 
maintaining  the  current  momentum  across  the  spectrum  of  government  decision  makers. 


12 


REPORT  DOCUMENTATION  PAGE 

Form  Approved 

0MB  No.  0704-0188 

Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathenng  and  maintaining  the  data  needed, 
and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information,  including  suggestions  for  reducing  this  burden,  to  Washington 
Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202-4302,  and  to  the  Office  of  Management  and  Budget,  Paperwork  Reduction 

Project  f0704-0188).  Washington,  DC  20503.  _ _ _ _ _ 

1.  AGENCY  USE  ONLY  (Leave  blank)  2.  REPORT  DATE  3. 

February  1999 

REPORT  TYPE  AND  DATES  COVERED 

Final 

4.  TITLE  AND  SUBTITLE 

Defining  Military  Experiments 

5.  FUNDING  NUMBERS 

DASW01-98-C-0067 

Task  T-I5-1524 

6.  AUTHOR(S) 

D.  Robert  Worley 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Institute  for  Defense  Analyses 

1801  N.  Beauregard  Street 

Alexandria,  VA  223 11-1772 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

IDA  Document  D-24I2 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Deputy  Director  for  Wargaming,  Simulation  and  Assessment 

Director  for  Force  Structure,  Resources,  and  Assessment  (J-8) 

The  Pentagon,  Room  1E965 

Washington,  DC  20318-8000 

10.  SPONSORING/MONITORING  AGENCY 
REPORT  NUMBER 

11.  SUPPLEMENTARY  NOTES 

12a.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  unlimited  distribution:  28  February  2000. 

12b.  DISTRIBUTION  CODE 

2A 

13.  ABSTRACT  (Maximum  200  words) 

DoD  recently  established  experimentation  as  a  new  military  mission.  Experimentation  supports  innovation,  transformation, 
and  the  revolution  in  military  affairs.  However,  experimentation,  in  a  military  context,  is  not  adequately  defined.  DoD 
documents,  congressional  speeches,  and  scholarly  literature  on  the  topic  have  been  examined  for  meaning  and  intent. 
Scientific  texts  on  both  basic  and  applied  research  have  been  examined  for  more  general  and  formal  definitions.  Four  models 
for  military  experiments  are  presented:  (1)  the  hypothesis-testing  laboratoiy  experiment  practiced  by  physical  scientists,  (2) 
heuristically  guided  experimentation  from  the  field  research  method  practiced  by  behavioral,  social,  and  life  scientists,  (3)  the 
goal-seeking  experiment  practiced  by  engineers  and  inventors  that  results  in  prototypes  or  patents,  and  (4)  the  performance¬ 
measuring  experiment  as  conducted  by  social  scientists  and  policy  analysts.  Of  the  four,  the  hypothesis-testing  experiment 
appears  to  be  serving  as  the  de  facto  model  for  military  experimentation,  yet  it  has  the  least  utility  in  fostering  innovation  and  a 
revolution  in  military  affairs. 


14.  SUBJECT  TERMS 

Experiment,  experimentation,  innovation,  transformation,  revolution  in  military  affairs,  RMA. 


15.  NO.  OF  PAGES 
26 


SECURITY  CLASSIFI¬ 
CATION  OF  REPORT 
UNCLASSIFIED 


18.  SECURITY  CLASSIFI¬ 
CATION  OF  THIS  PAGE 

UNCLASSIFIED 


19.  SECURITY  CLASSIFI¬ 
CATION  OF  ABSTRACT 

UNCLASSIFIED 


16. 

PRICE  CODE 

20. 

LIMITATION  OF 

ABSTRACT 

UL 

NSN  7540-01-280-5500 

Prescribed  by  ANSI  Std.  Z39-18 


Standard  Form  298  (Rev.  2-89) 


